Help:Special characters

From PyroGuide

Jump to: navigation, search

Template:H:h

Technical note: Due to technical limitations, some web browsers may not display some special characters in this article. Such characters may be rendered as boxes, question marks, or other replacement symbols, depending on your browser, operating system, and installed fonts. Even if you have ensured that your browser is interpreting the article as UTF-8 encoded and you have installed a font that supports a wide range of Unicode, such as w:Arial Unicode MS, Code2000, Code2001,TITUS Cyberbit Basic, Lucida Sans Unicode or one of the free software Unicode fonts, you may still need to use a different browser, as browser capabilities in this regard tend to vary.

Contents

[edit] Systems for character encoding

From MediaWiki 1.5, all projects use Unicode (UTF-8) character encoding.

Until the end of June 2005, when this new version came into use on Wikimedia projects, the English, Dutch, Danish, and Swedish Wikipedias used windows-1252 (they declared themselves to be ISO-8859-1 but in reality browsers treat the two as synonymous and the mediawiki software made no attempt to prevent use of stuff from windows-1252). Pre-upgrade wikitext in their databases remains stored in windows-1252 and is converted on load (some of it may also have been converted by gradual changes in the way history is stored). Edits made since the upgrade will be stored as UTF-8 in the database. This conversion on load process is invisible to users. It is also invisible to reusers as wikimedia now uses xml dumps rather than database dumps.

  • Unicode (UTF-8)
    • a variable number of bytes per character
    • special characters, including CJK characters, can be treated like normal ones; not only the webpage, but also the edit box shows the character; in addition it is possible to use the multi-character codes; they are not automatically converted in the edit box.
  • ISO 8859-1
    • one byte per character
    • special characters that are not available in the limited character set are stored in the form of a multi-character code; there are usually two or three equivalent representations, e.g. for the character € the named character reference € and the decimal character reference € and the hexadecimal character reference €. The edit box shows the entered code, the webpage the resulting character. Unavailable characters which are copied into the edit box are first displayed as the character, and automatically converted to their decimal codes on Preview or Save.
    • the most common special characters, such as é, are in the character set, so code like é, although allowed, is not needed.

Note that Special:Export exports using UTF-8 even if the database is encoded in ISO 8859-1, at least that was the case for the English Wikipedia, already when it used version 1.4.

To find out which character set applies in a project, use the browser's "View Source" feature and look for such as this: 

<meta http-equiv="Content-type" content="text/html; charset=iso-8859-1" />

or 

<meta http-equiv="Content-type" content="text/html; charset=utf-8" />

[edit] Editing

Many characters not in the repertoire of standard ASCII will be useful—even necessary—for wiki pages, especially for foreign language textbooks. This page contains recommendations for which characters are safe to use and how to use them. There are four ways to enter a non-ASCII character into the wikitext:

  • Use a link to a special character listed under the edit box to insert that character. Note however that some characters are not displayed in Internet Explorer:
    Image:Special characters under edit box, IE.png
    In some fonts, e.g. Arial, all the characters in this box are displayed, but it is not convenient for a user to have to switch fonts between webpages.
  • Enter the character directly from a foreign keyboard, or by cut and paste from a "character map" type application, or by some special means provided by the operating system or text editing application. On ISO-8859-1 wikis some browsers will change characters outside the charset of the wiki into html numeric character entities (see below).
  • Use an HTML named character entity reference like &agrave;. This is unambiguous even when the server does not announce the use of any special character set, and even when the character does not display properly on some browsers. However, it may cause difficulties with searches (see below).
  • Use an HTML numeric character entity reference like &#161;. Unfortunately some old browsers incorrectly interpret these as references to the native character set. It is, however, the only way to enter Unicode values for which there is no named entity, such as the Turkish letters. Note that because the code points 128 to 159 are unused in both ISO-8859-1 and Unicode, character references in that range such as &#131; are illegal and ambiguous, though they are commonly used by many web sites. (Note they are not technically unused, but they map to rare control codes that are illegal in HTML.) Also note that almost all browsers treat iso-8859-1 as windows-1252, which does have printable characters in that space, and they often found their way into article titles on en, which really caused confusion when trying to create interwiki links to said pages.

Generally speaking, Western European languages such as Spanish, French, and German pose few problems. For specific details about other languages, see: Help:Turkish characters and Help:Romanian characters. (More will be added to this list as contributors in other languages appear.)

For the purpose of searching, a word with a special character can best be written using the first method. If the second method is used a word like Odiliënberg can only be found by searching for Odili, euml and/or nberg; this is actually a bug that should be fixed—the entities should be folded into their raw character equivalents so all searches on them are equivalent. See also Help:Searching.

[edit] Esperanto

in edit boxin database and output
SS
SxŜ
SxxSx
SxxxŜx
SxxxxSxx
SxxxxxŜxx

Mediawiki installations configured for Esperanto use UTF-8 for storage and display. However when editing the text is converted to a form that is designed to be easier to edit with a standard keyboard.

The characters for which this applies are: Ĉ, Ĝ, Ĥ, Ĵ, Ŝ, Ŭ, ĉ, ĝ, ĥ, ĵ, ŝ, ŭ. you may enter these directly in the edit box if you have the facilities to do so. However when you edit the page again you will see them encoded as Sx. This form is referred to as "x-sistemo" or "x-kodo". In order to preserve round trip capability when one or more x's follow these characters or their non-accented forms (A, G, H, J, S, U, c, g, h, j, s, u), the number of x's in the edit box is double the number in the actual stored article text.

For example, the interlanguage link [[en:Luxury car]] to en:Luxury car has to be entered in the edit box as [[en:Luxxury car]] on eo:. This has caused problems with interwiki update bots in the past.

[edit] Browser issues

Some browsers are known to do nasty things to text in the edit box. Most commonly they convert it to an encoding native to the platform (whilst the NT line of Windows is internally UTF-16 it has a complete duplicate set of APIs in the Eindows ANSI code page and many older apps tend to use these, especially for things like edit boxes). Then they let the user edit it using a standard edit control and convert it back. The result is that any characters that do not exist in the encoding used for editing get replaced with something that does (often a question mark though at least one browser has been reported to actually transliterate text!).

[edit] IE for the Mac

This relatively common browser translates to mac-roman for the edit box with the result it munges most Unicode stuff (usually but not always by replacing them with a question mark). It also munges things that are in ISO-8859-1 but not mac-roman (specifically ¤ ¦ ¹ ² ³ ¼ ½ ¾ Ð × Ý Þ ð ý þ and the soft hyphen) so the problems it causes are not limited to Unicode wikis (though they tend to be much worse on Unicode wikis because they affect actual text and interwiki links rather than just fairly obscure symbols).

[edit] Netscape 4.x

Similar issues to IE Mac though the character set converted to and from will obviously not always be mac-roman.

[edit] Console browsers

Lynx links (in text mode) and W3M convert to the console character set (lynx actually using a transliteration engine) for editing and convert back on save. If the console character set is UTF-8 then these browsers are Unicode safe but if it isn't they aren't. With lynx and links a possible detection method would be to add another edit box to the login form but this won't work for w3m as it doesn't convert the text to the console character set until the user actually attempts to edit it.

[edit] The workaround

In database and edit
box for normal browsers		 In edit box
for bad browsers
œ&#x153;
&#x153;&#x0153;
&#x0153;&#x00153;
PG&G Company;

After en switched to utf-8 and interwiki bots started replacing html entities in interwikis with literal unicode text, edits that broke unicode characters became so common they could no longer be ignored. A workaround was developed to allow broken browsers to edit safely provided mediawiki knew they were broken.

Browsers listed in the setting $wgBrowserBlackList (a list of regexps that match against user agent strings) are supplied text for editing in a special form. Existing hexadecimal html entities in the page have an extra leading zero added, non-ascii characters that are stored in the wikitext are repreresented as hexadecimal html entities with no leading zeros.

Currently the default settings only have IE mac and a specific version of netscape 4.x for linux in the blacklist. Nevertheless it seems to have stopped most of the problem. Hopefully the default list will be expanded in future but that relies on getting someone with cvs access to commit the changes.

[edit] Viewing

Most current browsers have some level of Unicode support but some do it better than others. The most commonly encountered problem is that Internet Explorer relies on preconfigured font links in the registry rather than actually searching for a font that can display the character in question. This means that Internet Explorer often has to be forced to use particular fonts. On en there are a set of templates to do this. For example Template:Tlw for general Unicode text, Template:Tlw for polytonic Greek and Template:Tlw for the International Phonetic Alphabet. The stuff in Windows Glyph List 4 should be safe to use without such special measures.

<font face="Arial Unicode MS">...</font> may work, but only for people with that font.

[edit] Egyptian Hieroglyphs

E.g. <hiero>P2</hiero> gives <hiero>P2</hiero> See Help:WikiHiero syntax.

This is not dependent on browser capabilities, because it uses images on the servers.

Hieroglyphs could also be represented using Unicode, however, browser support for this is likely to be near non-existent.

[edit] Linking text with special characters

Many users have settings giving underlined links. When linking a special character, in some cases the result may be mistaken for another character with a different meaning:

Linking + − < > ⊂ ⊃ gives + − < > ⊂ ⊃ which may look like ± = ≤ ≥ ⊆ ⊇. In such cases one can better use a separate link:

There is less risk of confusion if more than one character is linked, e.g. x > 3.

[edit] See also

[edit] External links

Template:H:f

Retrieved from "http://www.pyroguide.com/index.php?title=Help:Special_characters"
Personal tools
pyroguide forum
pyroguide sponsors
pyroguide visitors